delay)delay over battery - without outliers)This document analyses the self-developed scheduler component in Android-based mobile devices as part of the SyMptOMS project. This sofware component tested here is called NativeScript Task Dispatcher, that’s a NativeScript plugin aimed to ease mobile app’s tasks definition and the execution of task workflows in background, regardless of whether the app is visible to the user or not. NativeScript Task Dispatcher is publicy available as open source software.
To create the PDF of the computational notebook you can run the following commands in a new R session. If you have problems rendering the PDF you can execute each chunk independently in RStudio.
This document does not install the required R packages by default. You can run the script install.R to install all required dependencies on a new R installation, or use install.packages(..) to install missing R packages.
The plots and tables use the packages ggplot2, knitr::kable() and kableExtra.
Required libraries and runtime environment description are as follows.
library(tidyverse)
library(kableExtra)
library(here)
# library(googledrive)
library(lubridate)
library(stringr)
library(scales)
# library(plotly)
library(grid)
library(gridBase)
library(gridExtra)
library(egg)
Log files are stored in shared gdrvie folder. This notebook downloads them locally in the folder data-raw.
| experiment | gsheets_name | gsheets_link |
|---|---|---|
| #1 | AD_BQ.csv | https://drive.google.com/open?id=1fP9r0S8ORa689yHRtfCtYQothLuGcCw3 |
| #1 | AD_NV.csv | https://drive.google.com/open?id=1l-vnaT-Smy0SegArIz5A108-IQ2_jmVV |
| #1 | AD_A1.csv | https://drive.google.com/open?id=1gzk1ezN5t5yBv2RnbznUqvalDx97eu3o |
| #1 | BA_H9.csv | https://drive.google.com/open?id=1BLXO4Rvz6ppWFJe5oN9Y0hKh__ypofxk |
| #1 | BA_MO.csv | https://drive.google.com/open?id=1RizGjKR8QLOdJna7qGPP7dYX9WWAysO4 |
| #2 | BA_BQ.csv | https://drive.google.com/open?id=1rfaooLZ0Up0gNx1SCA92TV1JFxcLU0sk |
| #2 | BA_NV.csv | https://drive.google.com/open?id=1_KIXX5FGXv7MO37QbUDnd_KGqiGihiMI |
| #2 | BA_A1.csv | https://drive.google.com/open?id=1LezrtyB9i4K7cQsmI78E0v_KaOyhZcXI |
| #2 | AD_H9.csv | https://drive.google.com/open?id=1ISjsfSLZt6n9miryQtDr-tsZV9bfbCDT |
| #2 | AD_MO.csv | https://drive.google.com/open?id=1bx2RVNz05qVKGr-32GI-ZIOKQWlqKBPz |
Raw data files are cleaned and merged together into a single file stored in the folder data. New fields such as dates breakdown are computed too.
Key Variables:
quantitative (interval)
plan_date (datetime): task planning time.exec_date (datetime): task execution time.quantitative (ratio)
delay (numeric): execution delay in seconds ((exec_date - plan_date) - 60).categorital (ordinal)
battery(numeric): battery levelcategorical (nominal)
device_id/device_name: device identifier/nameexp_id: experiment identifierscheduler: scheduler type| # exp | Type scheduler | Device | # records | Start | End | Min | Bottom whisker | 1st Qu. | Median | Mean | 3rd Qu. | Top whisker | Max |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #1 | Advanced | BQ Aquaris V | 21391 | 2020-03-30 12:15:03 | 2020-04-14 13:07:57 | -48.591 | 0.235 | 0.546 | 0.651 | 0.769 | 0.754 | 1.065 | 61.204 |
| Nvidia Shield Tablet | 21453 | 2020-03-30 12:15:03 | 2020-04-14 13:08:14 | -52.811 | -0.882 | 0.052 | 0.241 | 0.673 | 1.043 | 2.518 | 103.849 | ||
| Xiaomi Mi A1 | 21465 | 2020-03-30 12:23:27 | 2020-04-14 13:07:50 | -60.925 | -0.840 | 0.088 | 0.269 | 0.493 | 0.838 | 1.945 | 68.872 | ||
| Basic | Honor 9 | 21574 | 2020-03-30 12:00:03 | 2020-04-14 13:01:35 | -0.824 | -0.094 | 0.068 | 0.087 | 0.311 | 0.187 | 0.364 | 295.963 | |
| Motorola Moto G | 21619 | 2020-03-30 12:00:07 | 2020-04-14 13:01:16 | -0.183 | -0.183 | 0.062 | 0.100 | 0.875 | 0.289 | 0.629 | 60.896 | ||
| #2 | Advanced | Honor 9 | 19377 | 2020-04-15 11:00:02 | 2020-04-29 12:59:15 | -59.535 | -0.423 | 0.167 | 0.225 | 0.591 | 0.842 | 1.853 | 93.032 |
| Motorola Moto G | 19735 | 2020-04-16 12:09:53 | 2020-04-30 12:58:44 | -30.166 | -0.212 | 0.117 | 0.167 | 2.092 | 0.782 | 1.778 | 126.487 | ||
| Basic | BQ Aquaris V | 20146 | 2020-04-15 11:16:26 | 2020-04-29 13:00:20 | -0.111 | 0.065 | 0.280 | 0.297 | 0.390 | 0.424 | 0.639 | 34.412 | |
| Nvidia Shield Tablet | 3528 | 2020-04-15 11:16:18 | 2020-04-17 22:27:06 | -0.871 | -0.871 | 0.021 | 0.122 | 0.431 | 1.007 | 2.085 | 4.841 | ||
| Xiaomi Mi A1 | 10832 | 2020-04-15 11:16:23 | 2020-04-23 00:52:00 | -0.905 | -0.661 | 0.066 | 0.302 | 0.405 | 0.614 | 1.416 | 108.035 |
Observations on the above table :
181.120 records in total.
50.6% (91.582/181.120) is the interquartile range or IQR, i.e. the middle 50% of the data.
39.6% (71.769/181.120) represent the reasonable extremes of the data, i.e. these whisker values are the minimum and maximum values that do not exceed 1.5 × IQR.
9.8% (17.769/181.120) are outliers, i.e. values beyond the top and bottom whiskers.
Nvidia Shield Tablet has stopped triggering alarms on 2020-04-17 22:27:06 when running the basic scheduler. Since there’s no watchdog in the basic version, this device is not going to trigger alarms anymore. Died!
Xiaomi Mi A1 has stopped triggering alarms on 2020-04-23 00:52:00 when running the basic scheduler. An explanation may be that Huawei/Honor smartphones on Android 9+ have a Launch Records where the OS records how much times an app has been launched by itself, so the OS has the possibility to detect this kind of apps. In these smartphones, Huawei also introduced a new task killer (Power Genius) which kills everything that is not whitelisted by Huawei, and does not give users any configuration options. Therefore, the Power Genius uses the information from the Launch Records to kill the scheduled alarms
In mental health interventions, especially in EMI, the predictibility of the execution of planned actions is mostly preferred since it means notifications and/or alerts are delivered to patients at the right moment. It is inconceivable, for example, that a patient gets a notification (e.g. action, alert, psyco content, etc.) 2 minutes later than the event (e.g. passing nearby a gambling hall) that triggered the action; the patient could be dozens of meters away from the place of interest or, even worse, walked into that no-desirable place. Therefore, closing both the temporal and spatial gaps between the triggered event and the response to the event is critical in mental health EMIs to ensure that time and space/place are in synchrony with respect to the action. If action-time-place(space) are not in synchrony, the intervetion becomes far less effective. Predictibiliy is then a necessary (but not sufficient) proxy for effectiveness.
Given that average response time (delay) is similar in all devices (< 1 second), outliers are a good candidate to study reliability/predictibility. In other words, if a scheduler shows fewer outliers (in terms of delay), it would mean that planned actions are more stable and predictable over time/space and, consequently, that scheduler does a good job in term of effectiveness. On the other hand, a high number of and/or frequent outliers would signal lack of synchrony between space/place and time dimensions in an intervention, resulting in a less predictable scheduler for EMI/A.
| Device | # exp | Type scheduler | # outliers | # outliers positive | # outliers negative |
|---|---|---|---|---|---|
| BQ Aquaris V | #1 | Advanced | 1646 | 925 | 721 |
| BQ Aquaris V | #2 | Basic | 2996 | 2499 | 497 |
| Honor 9 | #2 | Advanced | 307 | 209 | 98 |
| Honor 9 | #1 | Basic | 5033 | 5018 | 15 |
| Motorola Moto G | #2 | Advanced | 2861 | 2852 | 9 |
| Motorola Moto G | #1 | Basic | 4377 | 4377 | NA |
| Nvidia Shield Tablet | #1 | Advanced | 294 | 285 | 9 |
| Nvidia Shield Tablet | #2 | Basic | 3 | 3 | NA |
| Xiaomi Mi A1 | #1 | Advanced | 224 | 121 | 103 |
| Xiaomi Mi A1 | #2 | Basic | 28 | 27 | 1 |
| scheduler | n_total |
|---|---|
| Advanced | 5332 |
| Basic | 12437 |
Number of outliers: 4642
Number of outliers (top range, beyond top whisker): 3424 - 73.8% (3.424/4.642).
Number of outliers (bottom range, beyond bottom whisker): 1218 - 26.2% (1.218/4.642).
Number of outliers: 297
Number of outliers (top range, beyond top whisker): 288 - 97% (288/297).
Number of outliers (bottom range, beyond bottom whisker): 9 - 3% (9/297).
Number of outliers: 252
Number of outliers (top range, beyond top whisker): 148 - 58.7% (148/252).
Number of outliers (bottom range, beyond bottom whisker): 104 - 41.3% (104/252).
Number of outliers: 5340
Number of outliers (top range, beyond top whisker): 5227 - 97.9% (5.227/5.340).
Number of outliers (bottom range, beyond bottom whisker): 113 - 2.1% (113/5.340).
Number of outliers: 7238
Number of outliers (top range, beyond top whisker): 7229 - 99.9% (7.229/7.238).
Number of outliers (bottom range, beyond bottom whisker): 9 - 0.1% (9/7.238).
| scheduler | device_name | mean | median |
|---|---|---|---|
| Advanced | BQ Aquaris V | 2.1539143 | 1.0830 |
| Advanced | Honor 9 | 9.5330423 | 3.2580 |
| Advanced | Motorola Moto G | 12.7074159 | 9.0550 |
| Advanced | Nvidia Shield Tablet | 13.6128095 | 10.9860 |
| Advanced | Xiaomi Mi A1 | 5.0634241 | 8.5705 |
| Basic | BQ Aquaris V | 0.7612009 | 0.9000 |
| Basic | Honor 9 | 1.0521444 | 1.0180 |
| Basic | Motorola Moto G | 3.8990078 | 1.5000 |
| Basic | Nvidia Shield Tablet | 3.4696667 | 2.8620 |
| Basic | Xiaomi Mi A1 | 11.1098214 | 1.6820 |
| scheduler | mean | median |
|---|---|---|
| Advanced | 8.995551 | 4.9965 |
| Basic | 2.007192 | 1.0030 |
In response to the question Which one of the two schedulers is more reliable?, we observe the following statements based on the metric delay.
Feasibility on the execution of schedulers:
Basic scheduler stopped running in 2 (out of 5) devices.
Advanced scheduler completed the experiments on all devices (Note: to clarify issue with Honor 9)
Number of outliers:
The number of outliers in the experiments with the basic scheduler is more than double that of the experiments with the advanced scheduler. Therefore,the advanced scheduler beats the basic one in terms of number of outliers.
Honor 9 shows a remarkly difference between the two schedulers. This is importnat because this device was used regularly while runing the experiments, demonstrated by the distinct distribution of outliers at nigthtime and daytime.
Majority of outliers are positive (beyond top whisker). Negative values (beyond bottom whisker) should be interpreted as wrong values due to planned timestamps mistakenly recorded during data collection.
Frequency of outliers:
Centrality measurements of outliers:
delay)We assume that higher accuracy equals to lower delay. In Q1, we compete delayas execution timestamp(i) - planned timestamp(i), being i each time steps (roughly 1 minute). Here, we compute other types of delays to make sure that the scheduler works as expected. Therefore, wecompute the following timestamps differences:
Consecutive planning timestamps, planned timestamp(i-1) - planned timestamp(i), in order to detect wrong planned timestamp values
Consecutive execution timestamps, execution timestamp(i-1) - execution timestamp(i), to detect execution timestamps between consecutive measurements
Overhead, execution timestamp(i) - planning timestamp(i+1), to explore framework overhead between consecutive measurements
We calculate planned timestamp(i-1) - planned timestamp(i) to detect wrong planning timestamp values. During data collection, chances are that some planned timestamps are not correctly registered, creating duplicated timestamps, suggested by the presence of negative values of delay. The new variable/metric is called plan_date_diff.
| # exp | Type scheduler | Device | # records | Start | End | Min | Bottom whisker | 1st Qu. | Median | Mean | 3rd Qu. | Top whisker | Max |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #1 | Advanced | BQ Aquaris V | 21391 | 2020-03-30 12:15:03 | 2020-04-14 13:07:57 | 0 | 60.159 | 60.463 | 60.565 | 60.735 | 60.666 | 60.970 | 1323.965 |
| Nvidia Shield Tablet | 21453 | 2020-03-30 12:15:03 | 2020-04-14 13:08:14 | 0 | 58.629 | 60.012 | 60.179 | 60.560 | 60.980 | 62.415 | 191.099 | ||
| Xiaomi Mi A1 | 21465 | 2020-03-30 12:23:27 | 2020-04-14 13:07:50 | 0 | 58.936 | 60.039 | 60.224 | 60.501 | 60.796 | 61.887 | 628.508 | ||
| Basic | Honor 9 | 21574 | 2020-03-30 12:00:03 | 2020-04-14 13:01:35 | 0 | 59.907 | 60.014 | 60.025 | 60.243 | 60.103 | 60.235 | 296.462 | |
| Motorola Moto G | 21619 | 2020-03-30 12:00:07 | 2020-04-14 13:01:16 | 0 | 59.980 | 60.008 | 60.016 | 60.117 | 60.027 | 60.055 | 120.719 | ||
| #2 | Advanced | Honor 9 | 19377 | 2020-04-15 11:00:02 | 2020-04-29 12:59:15 | 0 | 59.426 | 60.063 | 60.109 | 62.794 | 60.738 | 61.748 | 44935.248 |
| Motorola Moto G | 19735 | 2020-04-16 12:09:53 | 2020-04-30 12:58:44 | 0 | 59.733 | 60.032 | 60.064 | 61.441 | 60.377 | 60.895 | 186.625 | ||
| Basic | BQ Aquaris V | 20146 | 2020-04-15 11:16:26 | 2020-04-29 13:00:20 | 0 | 60.045 | 60.247 | 60.262 | 60.351 | 60.385 | 60.592 | 94.369 | |
| Nvidia Shield Tablet | 3528 | 2020-04-15 11:16:18 | 2020-04-17 22:27:06 | 0 | 59.113 | 60.002 | 60.099 | 60.388 | 60.979 | 62.029 | 64.777 | ||
| Xiaomi Mi A1 | 10832 | 2020-04-15 11:16:23 | 2020-04-23 00:52:00 | 0 | 59.285 | 60.009 | 60.269 | 60.352 | 60.579 | 61.429 | 168.009 |
| Device | Type scheduler | # outliers | # outliers positive | # outliers negative |
|---|---|---|---|---|
| BQ Aquaris V | Advanced | 1698 | 888 | 810 |
| BQ Aquaris V | Basic | 3085 | 2517 | 568 |
| Honor 9 | Advanced | 455 | 263 | 192 |
| Honor 9 | Basic | 5066 | 5048 | 18 |
| Motorola Moto G | Advanced | 2589 | 2560 | 29 |
| Motorola Moto G | Basic | 3653 | 3287 | 366 |
| Nvidia Shield Tablet | Advanced | 197 | 181 | 16 |
| Nvidia Shield Tablet | Basic | 4 | 3 | 1 |
| Xiaomi Mi A1 | Advanced | 359 | 180 | 179 |
| Xiaomi Mi A1 | Basic | 21 | 19 | 2 |
We calculate execution timestamp(i-1) - execution timestamp(i) which gives us whether execution times between consecutive measurements (i.e. tasks execution) are accurate, i.e., tasks are executed as expected. The new variable/metric is called delay2 (substracting 60 to get values around zero).
delay over battery - without outliers)Higher reliability = higher percentage of alarms executed / expected alarms for the experiment time frame
Distribution of data based on a five number summary (“minimum”, first quartile (Q1), median, third quartile (Q3), and “maximum”).
Basic statistics applied to delay and device_name. Given a device, how do the box plots change on each scheduler?